{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Canary Roll Out of ML Models with Seldon and Istio\n",
    "\n",
    "This folder provides resources to illustrate how to do a canary\troll out of one\tMNIST model to another using the canary\tpattern where a small amount\tof traffic is sent to the new model to validate\tit before sending all traffic to the new model.\n",
    "\n",
    "We utilize two MNIST digit classification models. \n",
    "\n",
    " * Version 1 of the model using scikit-learn\n",
    " * Version 2 of the model using Tensorflow.\n",
    " \n",
    "After deploying Istio and Seldon to a kubernetes cluster we will:\n",
    "\n",
    " * Deploy version 1 scikit-learn based model using seldon-core\n",
    " * Create an istio routing rule to direct all traffic to this version\n",
    " * Create a canary deployment with both version 1 and version 2 (the Tensorflow model)\n",
    " * Update the istio routing to send 10% of the traffic to version 2\n",
    " * Update the istio routing to send 100% of the traffic to version 2\n",
    " \n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup\n",
    "\n",
    "The steps below will install istio and seldon onto a GKE Cluster. If you wish to use your own setup then you need to ensure\n",
    "\n",
    " * You allow istio egress to the internet as the load test downloads MNIST images\n",
    " * Ensure you give your user cluster-admin privledges\n",
    " * Install seldon into a namespace seldon\n",
    " \n",
    "To follow the steps below you will need:\n",
    " \n",
    "  * a Google project running a K8S cluster\n",
    "  * gcloud and kubectl installed with kubectl authorized for your cluster\n",
    "  * [istio download](https://github.com/istio/istio/releases/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup Environment Variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set this to your istio download folder, e.g. \n",
    "# %env ISTIO_HOME=~/istio-1.0.0\n",
    "%env ISTIO_HOME=<ISTIO-HOME>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set this to your GKE Zone for your cluster, e.g.\n",
    "# %env ZONE=europe-west3-b\n",
    "%env ZONE=<ZONE>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set this to the GCP Project within which your cluster is running\n",
    "%env PROJECT=<PROJECT>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set this to the name of your kubernetes cluster\n",
    "%env CLUSTER_NAME=<CLUSTER>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Determine CIDR ranges"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!gcloud container clusters describe ${CLUSTER_NAME} --zone ${ZONE} --project ${PROJECT} | grep -e clusterIpv4Cidr -e servicesIpv4Cidr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install Helm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl create clusterrolebinding my-cluster-admin-binding --clusterrole=cluster-admin --user=$(gcloud info --format=\"value(config.account)\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl create -f ${ISTIO_HOME}/install/kubernetes/helm/helm-service-account.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!helm init --service-account tiller"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install Istio\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Replace the CIDR 10.48.0.0/14\\,10.51.240.0/20 values with those you got above **\n",
    "\n",
    "For more details see [istio docs on egress](https://istio.io/docs/tasks/traffic-management/egress/#calling-external-services-directly)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!helm install ${ISTIO_HOME}/install/kubernetes/helm/istio --name istio --namespace istio-system \\\n",
    "    --set global.proxy.includeIPRanges=\"10.60.0.0/14\\,10.63.240.0/20\" --set grafana.enabled=true"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To view the istio Grafana dashboard:\n",
    "```\n",
    "kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=grafana -o jsonpath='{.items[0].metadata.name}') 3000:3000\n",
    "```\n",
    "http://localhost:3000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install Seldon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl create namespace seldon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!helm install ../../../helm-charts/seldon-core-crd --name seldon-core-crd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!helm install ../../../helm-charts/seldon-core --name seldon-core --namespace seldon \\\n",
    "    --set ambassador.enabled=true"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!helm install ../../../helm-charts/seldon-core-analytics --name seldon-core-analytics \\\n",
    "    --set grafana_prom_admin_password=password \\\n",
    "    --set persistence.enabled=false \\\n",
    "    --namespace seldon"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To send requests to Ambassador ingress in another terminal run:\n",
    "    \n",
    "```\n",
    "kubectl port-forward $(kubectl get pods -n seldon -l service=ambassador -o jsonpath='{.items[0].metadata.name}') -n seldon 8002:8080\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To view the Seldon Grafana dashboard in another terminal run:\n",
    "\n",
    "```\n",
    "kubectl port-forward $(kubectl get pods -n seldon -l app=grafana-prom-server -o jsonpath='{.items[0].metadata.name}') -n seldon 3001:3000\n",
    "```\n",
    "\n",
    "Use user:admin and the password passed in above in the helm install.\n",
    "\n",
    "http://localhost:3001/dashboard/db/prediction-analytics?refresh=5s&orgId=1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl label namespace seldon istio-injection=enabled"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl config set-context $(kubectl config current-context) --namespace=seldon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import utils\n",
    "from visualizer import get_graph\n",
    "mnist = utils.download_mnist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Launch Version 1 Model\n",
    "A sklearn MNIST model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pygmentize mnist_v1.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "get_graph(\"mnist_v1.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl apply -f mnist_v1.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Wait until new pods are running **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl get pods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "utils.predict_rest_mnist(mnist,\"mnist-classifier\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Start a Load Test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl label nodes $(kubectl get nodes -o jsonpath='{.items[0].metadata.name}') role=locust"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!helm install seldon-core-loadtesting --name loadtest  \\\n",
    "    --namespace seldon \\\n",
    "    --repo https://storage.googleapis.com/seldon-charts \\\n",
    "    --set locust.script=mnist_rest_locust.py \\\n",
    "    --set locust.host=http://mnist-deployment:8000 \\\n",
    "    --set oauth.enabled=false \\\n",
    "    --set oauth.key=oauth-key \\\n",
    "    --set oauth.secret=oauth-secret \\\n",
    "    --set locust.hatchRate=1 \\\n",
    "    --set locust.clients=1 \\\n",
    "    --set loadtest.sendFeedback=1 \\\n",
    "    --set locust.minWait=0 \\\n",
    "    --set locust.maxWait=0 \\\n",
    "    --set replicaCount=1 \\\n",
    "    --set data.size=784\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup Version 1 Istio Routing\n",
    "Just route to version 1 of our model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pygmentize istio_canary_v1.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!istioctl create -f istio_canary_v1.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Launch Version 2 of Model\n",
    "Version 2 is a Tensorflow Deep Learning Model for MNIST. Hopefully, better..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pygmentize mnist_v2.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "get_graph(\"mnist_v2.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl apply -f mnist_v2.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Wait until new pods are running **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl get pods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "utils.predict_rest_mnist(mnist,\"mnist-classifier\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup Canary Routing\n",
    "Let's be cautious and route 30% of traffic to version 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pygmentize istio_canary_v2.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!istioctl replace -f istio_canary_v2.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup Routing to Version 2\n",
    "After testing we are happy and want to switch 100% of traffic to version 2 our Tensorflow model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pygmentize istio_canary_v3.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!istioctl replace -f istio_canary_v3.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "utils.predict_rest_mnist(mnist,\"mnist-classifier\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Switch to final model\n",
    "Finally, lets save resources and remove version 1 of our model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pygmentize mnist_v3.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "get_graph(\"mnist_v3.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl apply -f mnist_v3.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Wait until pods ready**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl get pods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "utils.predict_rest_mnist(mnist,\"mnist-classifier\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
